Canofy customer onboarding SOP

Standard procedure for bringing a new customer onto a dedicated Canofy (Frappe) production instance on AWS, end to end, with SSM-only server access. Every step is written with the real values from the first customer onboarded this way, Belimo Automation India Pvt. Ltd. (code BEL). Replace them with the new customer's values from the variables table in section 0.3.

0. Before you start

0.1 How to read this SOP

Each step starts with a tag that tells you where to run it:

AWS console AWS CloudShell Server via SSM, as ssm-user Server via SSM, as frappe OCR server, as ubuntu GoDaddy Microsoft Entra Canofy in the browser GitHub

  • Commands are written one per box with a copy button. Paste one box at a time.
  • The SSM browser terminal can merge pasted multi-line text. This SOP therefore never uses heredocs (<<EOF): every file is written with a single-line command, or through S3.
  • Lines that contain credentials start with a space, so they aren't saved in shell history.
  • Screenshots come from the Belimo onboarding. Account IDs, tenant/client IDs and secret values are masked.

Note: Never paste passwords, client secrets, tokens or the OCR encryption key into chat, tickets, email or this document. Store them only in the password manager.

0.2 Access you need

Access Used for Belimo note
AWS console in ap-south-1 with IAM, EC2, S3, DLM and Systems Manager rights Phases 1–9 Operator account
GoDaddy access to canofy.tech DNS Phase 10 Needs manager access
GitHub read access to arnk-canofy/canofy_TCS, arnk-canofy/canofy_ddit, arnk-canofy/priv-frappe-s3-file-storage and the customer app repo Phase 16 Collaborator only, so a temporary classic token was used
The customer's Microsoft 365 mailbox login (no-reply-<code>@canofy.tech) Phase 24 The tenant allows normal users to register apps
SSM access to the OCR server (user ubuntu) Phase 25
A reference production server (e.g. MIL) Phase 15 Used to read versions and app remotes
Password manager Throughout

0.3 Customer variables

Fill in this table first. Every later step uses these values.

Variable Rule Belimo value
Customer legal name Belimo Automation India Private Limited
Customer code 3 letters, upper case BEL
Site / sub-domain <code>.canofy.tech (lower case) bel.canofy.tech
EC2 Name tag #LIVE Production <Client Name> [<CODE>] #LIVE Production Belimo Automation India [BEL]
IAM role EC2-Canofy-<CODE>-SSM-Prod EC2-Canofy-BEL-SSM-Prod
IAM S3 policy S3-<CODE> S3-BEL
S3 bucket <code>.canofy.tech (lower case) bel.canofy.tech
S3 folder (app) <CODE>, must match what OCR expects BEL
Instance type SOP default t3a.large t3a.medium (accepted deviation)
Root disk 40 GB recommended 20 GB (expansion pending)
Elastic IP allocated in phase 5 3.6.128.118
Private IP shown after launch 172.31.10.27
Mailbox no-reply-<code>@canofy.tech no-reply-bel@canofy.tech
Entra app Canofy-<CODE> Canofy-BEL
Customer app repo canofytech/canofy_custom_BEL (app custom_belimo)
OCR server shared 172.31.35.206:5000

0.4 Version lock

New customers must run the same versions as the reference production server. Fill this in during phase 15.

Component Version / branch Belimo
frappe pinned release tag v15.72.5
canofy_tcs branch development → 2.3.0 (4e82f42)
s3_file_storage branch main → 0.0.1 (163256d)
canofy_ddit branch, bench only (not installed on the site) main → 1.0.2 (782a128)
Customer app branch custom_belimo main → 1.2.3
Python / Node 3.11 / 20
MariaDB Ubuntu 22.04 default 10.6

1. Identity and access (IAM)

1.1 Create the S3 policy for the customer bucket

AWS console IAM → Policies → Create policy → JSON. Paste this, with the bucket name replaced:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketLevel",
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::bel.canofy.tech"
    },
    {
      "Sid": "ObjectLevel",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::bel.canofy.tech/*"
    }
  ]
}

Name it S3-BEL, then Create policy. The bucket doesn't need to exist yet.

1.2 Create the EC2 role

AWS console IAM → Roles → Create role → Trusted entity AWS service, use case EC2. Attach:

  • AmazonSSMManagedInstanceCore (or your equivalent SSM policy); this is mandatory, because SSM is the only way in;
  • S3-BEL.

Name it EC2-Canofy-BEL-SSM-Prod, add tags Customer=BEL and Environment=Production, then create it. An EC2 instance can only have one role, which is why both policies go on the same one.

Belimo: the role was created with a custom SSM policy equivalent to AmazonSSMManagedInstanceCore, plus S3-BEL.

2. Launch the EC2 instance

2.1 Choose the AMI

AWS console EC2 → Launch instance → AMI search.

  • Preferred: Community AMIs, search ubuntu-jammy-22.04-amd64-server, owner 099720109477 (Canonical), 64-bit x86, newest date.
  • Alternative: a Canofy golden AMI (phase 17), once one exists.
  • Avoid the Marketplace listing shown below. It works, but it adds a Marketplace product code to every AMI made from the instance.

You can also get the current Canonical AMI ID from CloudShell:

AWS CloudShell

aws ssm get-parameter --name /aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id --region ap-south-1 --query Parameter.Value --output text
Figure 1 SS-05a: the Marketplace "Subscribe" page for Ubuntu 22.04. Close it and use the Canonical Community AMI instead.

2.2 Launch settings

Setting Value
Name #LIVE Production Belimo Automation India (add [BEL] to match the volume naming)
Tags Customer=BEL, Environment=Production, Owner=Canofy
Instance type t3a.large (SOP). Belimo uses t3a.medium.
Key pair Proceed without a key pair
Network production VPC (vpc-cadaeca2), public subnet, auto-assign public IP on
Security group the customer security group (phase 3)
Storage gp3, encrypted (KMS aws/ebs), 40 GB recommended
IAM instance profile EC2-Canofy-BEL-SSM-Prod
Termination protection Enable
Stop protection Enable
Metadata V2 only (token required) recommended. Belimo kept "optional" for now.

Warning: IMDSv2 protects the instance-role credentials from SSRF attacks through the application. If you leave it optional, record it as a deviation and fix it before the ISO audit (Actions → Instance settings → Modify instance metadata options; no downtime).

2.3 Verify the instance

AWS console Open the instance → Details. Check: Running, instance type, IAM role, no key pair, IMDSv2 setting.

Figure 2 SS-06: instance summary after launch: Running, t3a.medium, role EC2-Canofy-BEL-SSM-Prod, auto-assigned public IP.

3. Security group

AWS console EC2 → instance → Security tab → open the security group → Inbound rules.

Type Port Source Keep?
HTTP 80 0.0.0.0/0 Yes: Let's Encrypt validation and the redirect to HTTPS
HTTPS 443 0.0.0.0/0 Yes: the application
SSH 22 any Delete. SSM doesn't need it.

Outbound: All traffic, 0.0.0.0/0 (needed for SSM, apt, GitHub and S3). If the instance is on the VPC's default group, create a dedicated group instead of editing the shared one.

Screenshot to capture, SS-04: Security group inbound rules showing only 80 and 443

4. EBS volume: encryption check and tags

AWS console Instance → Storage → click the volume ID.

  1. Check Encryption = Encrypted, KMS aws/ebs, type gp3.
  2. Tags → Manage tags:
Key Value
Name #LIVE Production Belimo Automation India [BEL]
Customer BEL
Environment Production
Backup Daily

The Name value matters: the 60-day snapshot policy (phase 9) finds volumes by this tag.

Warning: you can't encrypt an existing volume in place. If it shows "Not encrypted", stop. Snapshot it, copy the snapshot with encryption, and replace the volume before installing anything.

Figure 3 SS-07: volume vol-0cc1d95a7394d274a: 20 GiB gp3, 3000 IOPS, Encrypted with aws/ebs. Note the Customer tag here still holds a wrong value, which was corrected afterwards.

5. Elastic IP

AWS console EC2 → Elastic IPs → Allocate. Tags: Name=EIP-BEL-PROD, Customer=BEL. Then Actions → Associate, choose the instance and its private IP, and click Associate.

Record the IP in the variables table. AWS charges for every public IPv4 address, so release it if the instance is ever terminated.

Figure 4 SS-08: the Elastic IP 3.6.128.118 now replaces the auto-assigned address.

6. First SSM connection

AWS console Systems Manager → Fleet Manager. The instance must show Online. Then EC2 → instance → Connect → Session Manager → Connect.

Server via SSM, as ssm-user

whoami
lsb_release -a
sudo snap services amazon-ssm-agent
df -h / && free -h
curl -s https://checkip.amazonaws.com

Expected: ssm-user, Ubuntu 22.04.x (jammy), agent enabled active, and the Elastic IP as the outbound address.

Figure 5 SS-10: first SSM session checks: ssm-user, Ubuntu 22.04.5, agent active, outbound IP 3.6.128.118.

7. S3 bucket

7.1 Create the bucket

AWS console S3 → Create bucket:

Setting Value
Region ap-south-1
Name bel.canofy.tech (bucket names must be lower case)
Object ownership ACLs disabled
Block all public access On (all four)
Versioning Enable
Default encryption SSE-S3, Bucket Key enabled
Tags Customer=BEL, Environment=Production (mind the capital C)

Note: bucket names with dots (bel.canofy.tech) can cause TLS certificate mismatches in some S3 clients, because of virtual-hosted addressing. They work with the AWS CLI, boto3 and the Canofy OCR service (SLC uses slc.canofy.tech). A dash-style name such as canofy-bel avoids the issue completely.

Figure 6 SS-11: bucket properties: versioning enabled, SSE-S3 with Bucket Key, SSE-C blocked. The tag key "customer" must be "Customer".

7.2 Allow HTTPS only

AWS console Bucket → Permissions → Bucket policy → Edit:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": ["arn:aws:s3:::bel.canofy.tech", "arn:aws:s3:::bel.canofy.tech/*"],
      "Condition": { "Bool": { "aws:SecureTransport": "false" } }
    }
  ]
}
Screenshot to capture, SS-12: Bucket Permissions tab with Block Public Access on and the policy saved

7.3 Lifecycle rules

AWS console Bucket → Management → Create lifecycle rule.

Rule 1, BEL-cleanup-old-versions: applies to all objects. Tick only:

  • Permanently delete noncurrent versions → 90 days;
  • Delete expired object delete markers, and Delete incomplete multipart uploads → 7 days.

Warning: never tick "Expire current versions" on an all-objects rule. It would delete the customer's live documents.

Rule 2, BEL-expire-db-backups (create it after phase 21): prefix filter _backups/. Expire current versions → 30 days; permanently delete noncurrent versions → 1 day.

Screenshot to capture, SS-14: Both lifecycle rules listed on the Management tab

7.4 Test bucket access from the server

Server via SSM, as ssm-user

sudo snap install aws-cli --classic
aws sts get-caller-identity
echo "ssm-test" > /tmp/t.txt && aws s3 cp /tmp/t.txt s3://bel.canofy.tech/BEL/_healthcheck.txt --region ap-south-1 && aws s3 ls s3://bel.canofy.tech/BEL/ --region ap-south-1 && aws s3 rm s3://bel.canofy.tech/BEL/_healthcheck.txt --region ap-south-1

Expected: assumed-role/EC2-Canofy-BEL-SSM-Prod/..., then upload:, the listing, and delete:, with no Access Denied.

Figure 7 SS-17: role identity and S3 upload, list and delete, with no access keys on the server.

8. Give the OCR server access to the bucket

AWS console IAM → Policies → OCR-production-S3 → Edit → JSON.

The policy has two statements: an Allow that lists the customer buckets, and a Deny with NotResource that blocks every bucket not listed. Add the new bucket to both, directly after an existing line that already ends with a comma:

"arn:aws:s3:::bel.canofy.tech",
"arn:aws:s3:::bel.canofy.tech/*",

Click Next → Save changes and wait for the green "updated" banner. Then confirm in CloudShell that the default version contains the bucket 4 times:

AWS CloudShell

ARN=$(aws iam list-policies --scope Local --query "Policies[?PolicyName=='OCR-production-S3'].Arn" --output text); VER=$(aws iam get-policy --policy-arn "$ARN" --query Policy.DefaultVersionId --output text); echo "$VER"; aws iam get-policy-version --policy-arn "$ARN" --version-id "$VER" --query PolicyVersion.Document --output json | grep -c "bel.canofy.tech"

Warning: in the Belimo run, the first edit was never saved (screenshot below, with no BEL lines). Always confirm with the count of 4, not with the editor view.

Figure 8 Before: the saved policy has no bel.canofy.tech lines, so OCR is blocked by the Deny statement.
Figure 9 SS-13: after: BEL lines 20–21 (Allow) and 40–41 (Deny/NotResource), "updated" banner, Errors 0.

Note: the editor flags "arn:aws:s3:::mil.canofy.tech/" (trailing slash) as redundant. That's a pre-existing MIL typo. Fix it as a separate change, by removing only the trailing slash and never the /* line.

Figure 10 The suggestion markers on lines 22–23 come from the MIL trailing-slash typo, not from the new entries.

9. Snapshot backups (Data Lifecycle Manager)

AWS console EC2 → Lifecycle Manager.

Figure 11 Existing policies: an enabled default 14-day policy for all volumes, and the enabled 60_day_production_instances policy.

Open 60_day_production_instances → Modify. Under target resource tags, add Key Name, Value exactly the volume's Name tag (#LIVE Production Belimo Automation India [BEL]), then save.

Warning: the key must be Name, as for the other customers. In the screenshot below, the BEL entry was first added with key BEL, which matches no volume. It was corrected.

Figure 12 SS-15: target tags of the 60-day policy. Every customer uses key "Name" with the volume name. The "BEL" key entry is the mistake to avoid.

The next day, check EC2 → Snapshots for a BEL-Prod snapshot tagged dlm:managed.

10. DNS record

GoDaddy My Products → canofy.tech → DNS → Add New Record: Type A, Name bel (not the full domain), Value 3.6.128.118, TTL 600 s. Check that no other bel record exists.

Server via SSM, as ssm-user

getent hosts bel.canofy.tech

Expected: 3.6.128.118. SSL (phase 20) can't be issued until this resolves.

Screenshot to capture, SS-16: GoDaddy DNS record list showing the bel A record

11. Server baseline

Server via SSM, as ssm-user Start a tmux session first, so a dropped browser tab doesn't kill long-running commands. If you get disconnected, reattach with tmux attach -t setup.

sudo apt install -y tmux && tmux new -s setup
sudo apt update && sudo NEEDRESTART_MODE=a apt -y upgrade
sudo hostnamectl set-hostname bel-prod && echo "127.0.1.1 bel-prod" | sudo tee -a /etc/hosts && echo "preserve_hostname: true" | sudo tee /etc/cloud/cloud.cfg.d/99-hostname.cfg && sudo timedatectl set-timezone Asia/Kolkata
sudo fallocate -l 4G /swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile && echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab && echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf && sudo sysctl -p /etc/sysctl.d/99-swappiness.conf
sudo reboot

Reconnect, then check:

hostname; timedatectl | grep "Time zone"; free -h; swapon --show; df -h /
Figure 13 SS-11a: hostname bel-prod, timezone Asia/Kolkata (IST).
Figure 14 SS-11b: 4.0 GB swap active on /swapfile after the reboot.

12. System packages, frappe user and certbot

Server via SSM, as ssm-user

sudo adduser --disabled-password --gecos "" frappe && sudo usermod -aG sudo frappe && sudo chmod 755 /home/frappe
sudo NEEDRESTART_MODE=a apt install -y git curl software-properties-common build-essential pkg-config mariadb-server mariadb-client libmysqlclient-dev redis-server nginx supervisor xvfb libfontconfig fontconfig python3-dev python3-pip python3-setuptools python3-venv cron
sudo add-apt-repository -y ppa:deadsnakes/ppa && sudo apt update && sudo NEEDRESTART_MODE=a apt install -y python3.11 python3.11-dev python3.11-venv
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - && sudo NEEDRESTART_MODE=a apt install -y nodejs && sudo npm install -g yarn
cd /tmp && wget https://github.com/wkhtmltopdf/packaging/releases/download/0.12.6.1-2/wkhtmltox_0.12.6.1-2.jammy_amd64.deb && sudo NEEDRESTART_MODE=a apt install -y ./wkhtmltox_0.12.6.1-2.jammy_amd64.deb && cd ~

Install certbot now. bench setup sudoers (which runs inside setup production) fails if certbot is missing.

sudo snap install --classic certbot && sudo ln -sf /snap/bin/certbot /usr/bin/certbot && certbot --version

Temporary passwordless sudo for frappe. The frappe user has no password (SSM only), and bench init calls sudo supervisorctl. Remove this file at go-live (phase 26).

echo "frappe ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/90-frappe-setup && sudo chmod 440 /etc/sudoers.d/90-frappe-setup && sudo visudo -c

Check the versions:

id frappe; python3.11 --version; node -v; yarn -v; mariadb --version; redis-server --version; nginx -v; wkhtmltopdf --version
Figure 15 SS-SW-01: frappe in the sudo group, Python 3.11.16, Node v20.20.2, yarn 1.22.22, MariaDB 10.6.23, Redis 6.0.16, nginx 1.18.0, wkhtmltopdf 0.12.6.1 with patched Qt.

13. MariaDB

Server via SSM, as ssm-user Write the config with one line (heredocs get mangled in the SSM terminal). It includes the two settings Canofy's wide tables need: innodb_strict_mode = OFF and innodb_default_row_format = dynamic.

sudo bash -c 'printf "%s\n" "[mysqld]" "character-set-client-handshake = FALSE" "character-set-server = utf8mb4" "collation-server = utf8mb4_unicode_ci" "bind-address = 127.0.0.1" "innodb_buffer_pool_size = 1G" "innodb_log_file_size = 256M" "max_allowed_packet = 256M" "innodb_strict_mode = OFF" "innodb_default_row_format = dynamic" "" "[mysql]" "default-character-set = utf8mb4" > /etc/mysql/mariadb.conf.d/99-frappe.cnf'
sudo cat -A /etc/mysql/mariadb.conf.d/99-frappe.cnf

Every line must end in $, with no ^M and no merged lines. Then restart:

sudo systemctl restart mariadb && sudo systemctl status mariadb --no-pager | head -5
Figure 16 SS-DB-01: the config written with printf (one setting per line) and MariaDB active (running). The first lines show the earlier merged "256M[mysql]" that broke the service.

Secure the root account. Save the password in the password manager as <CODE>-Prod MariaDB root, and avoid ' " \ $ and spaces in it.

sudo mysql_secure_installation

Answers: Enter (no current password) → n (unix_socket) → Y (set a root password) → Y → Y → Y → Y.

sudo systemctl disable --now redis-server
sudo mysql -e "SHOW VARIABLES WHERE Variable_name IN ('character_set_server','collation_server','innodb_buffer_pool_size','bind_address','innodb_strict_mode');"
mysql -u root -p -e "SELECT VERSION();"
Figure 17 SS-DB-02: secure installation done; utf8mb4, 127.0.0.1, 1 GB buffer pool; root password login works; system Redis inactive.

14. Install bench and initialise Frappe

Server via SSM, as ssm-user

sudo -H pip3 install frappe-bench && bench --version

Server via SSM, as frappe Switch user inside tmux:

sudo -iu frappe
bench init --frappe-branch v15.72.5 --python python3.11 frappe-bench

Wait for SUCCESS: Bench frappe-bench initialized. This takes 10–20 minutes. Then add the missing setuptools package, without which bench new-site fails with No module named 'pkg_resources':

cd ~/frappe-bench && uv pip install --python ./env/bin/python "setuptools<81" && ./env/bin/python -c "import pkg_resources; print('pkg_resources OK')"
bench version && ./env/bin/python --version && df -h / && free -h
Figure 18 SS-19: bench initialised, frappe 15.72.5, Python 3.11.16.

15. Read the version lock from a reference server

Server via SSM, as frappe On an existing production server (Belimo used MIL), as frappe, run:

cd ~/frappe-bench && bench version && ./env/bin/python --version && node -v && cat sites/apps.txt
Figure 19 Reference versions on MIL: frappe 15.72.5, canofy_tcs 2.2.1, canofy_ddit 1.0.2, s3_file_storage 0.0.1, custom_mil_power (MIL only), Python 3.10.12, Node 18.20.8.

For each app, record the remote, branch and commit. Switch user first, and then run one app per line. Running it through sudo -iu frappe bash -c '…' loses the loop variable.

cd apps/canofy_tcs && git remote -v && git branch --show-current && git log -1 --oneline && cd ~/frappe-bench

Repeat for canofy_ddit and s3_file_storage.

Figure 20 App remotes on MIL: canofy_TCS on development (e5c1cf96), canofy_ddit main (v1.0.1 tag, 782a128), priv-frappe-s3-file-storage main (163256d).

16. Download the Canofy apps

The repos are private. Preferred: read-only deploy keys (one per repo), added by a repo admin. If you only have collaborator access: use a short-lived classic token that is never written to disk (below).

Figure 21 Without credentials, git reports "could not read Username", which means the repo is private.

16.1 Temporary classic token (collaborator access)

GitHub Settings → Developer settings → Tokens (classic) → Generate: note <CODE>-setup-temp, expiry 7 days, scope repo. If the organisation uses SSO, click Configure SSO → Authorize.

Server via SSM, as frappe In ~/frappe-bench:

read -s GH_TOKEN; echo "length: ${#GH_TOKEN}"
export GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.https://github.com/.extraheader GIT_CONFIG_VALUE_0="AUTHORIZATION: basic $(printf 'x-access-token:%s' "$GH_TOKEN" | base64 -w0)"
git ls-remote https://github.com/arnk-canofy/canofy_TCS.git HEAD
bench get-app canofy_ddit https://github.com/arnk-canofy/canofy_ddit.git --branch main
bench get-app s3_file_storage https://github.com/arnk-canofy/priv-frappe-s3-file-storage.git --branch main
bench get-app canofy_tcs https://github.com/arnk-canofy/canofy_TCS.git --branch development
unset GH_TOKEN GIT_CONFIG_COUNT GIT_CONFIG_KEY_0 GIT_CONFIG_VALUE_0

The customer app is downloaded after the golden-AMI checkpoint (phase 17), in phase 18.

supervisor group frappe: ERROR (no such group) warnings after get-app are expected at this stage.

for a in canofy_ddit s3_file_storage canofy_tcs; do echo "== $a"; git -C apps/$a branch --show-current; git -C apps/$a log -1 --oneline; git -C apps/$a remote get-url upstream; done
grep -rl "ghp_" ~/.gitconfig ~/.git-credentials apps/*/.git/config 2>/dev/null; echo "token check done"

Then delete the token on GitHub.

Figure 22 SS-21: versions and commits after download. canofy_tcs on development came in at 2.3.0, newer than MIL's 2.2.1, which is expected when following a branch.

16.2 Deploy keys (permanent)

Server via SSM, as frappe Create one key per repo, plus an SSH config that tells git which key belongs to which repo:

mkdir -p ~/.ssh && chmod 700 ~/.ssh && for k in canofy_tcs canofy_ddit s3_file_storage; do ssh-keygen -t ed25519 -N "" -C "bel-prod $k" -f ~/.ssh/deploy_$k; done
printf '%s\n' 'Host gh-canofy-tcs' '  HostName github.com' '  User git' '  IdentityFile ~/.ssh/deploy_canofy_tcs' '  IdentitiesOnly yes' '' 'Host gh-canofy-ddit' '  HostName github.com' '  User git' '  IdentityFile ~/.ssh/deploy_canofy_ddit' '  IdentitiesOnly yes' '' 'Host gh-s3-file-storage' '  HostName github.com' '  User git' '  IdentityFile ~/.ssh/deploy_s3_file_storage' '  IdentitiesOnly yes' > ~/.ssh/config && chmod 600 ~/.ssh/config

A repo admin adds each .pub key under Repo → Settings → Deploy keys (read-only). Collaborators don't see this page; the Actions secrets page below is the wrong place.

Figure 23 Settings → Secrets and variables → Actions is NOT where deploy keys go. Deploy keys need admin rights on the repo.

Then point the remotes at the keys:

git -C apps/canofy_tcs remote set-url upstream git@gh-canofy-tcs:arnk-canofy/canofy_TCS.git

Do the same for the other repos. Test each key with ssh -T gh-canofy-tcs.

Belimo: keys were generated, but an admin hasn't added them yet. That's an open item.

17. Golden AMI checkpoint

This is the point to capture a reusable image: everything common to all customers is installed, and nothing customer-specific exists yet (no site, SSL, DNS or customer app).

17.1 Raw image of the source server

Server via SSM, as frappe Confirm that no site exists yet (the folder should only contain apps.json, apps.txt, assets and common_site_config.json):

ls ~/frappe-bench/sites/

AWS console Instance → Actions → Image and templates → Create image: name TMP-<CODE>-raw-<YYYYMMDD>, Reboot ticked, tag Purpose=temp. Wait until it shows Available.

Belimo: TMP-BEL-raw-20260924 was created. Parts 17.2–17.4 are still pending.

17.2 Builder instance and clean-up

AWS console Launch TMP-AMI-Builder from the raw image (no key pair, auto-assign public IP, no protections, no Customer/Backup tags).

Server via SSM, as ssm-user On the builder, first make sure you're not on the production server:

cat /var/lib/cloud/data/instance-id

It must not be the production instance ID (BEL: i-0be6b206aecd2807e). Then:

sudo rm -f /home/frappe/.ssh/deploy_* /home/frappe/.ssh/config /home/frappe/.ssh/known_hosts && sudo rm -f /etc/cloud/cloud.cfg.d/99-hostname.cfg && sudo sed -i '/bel-prod/d' /etc/hosts
sudo rm -f /home/frappe/.bash_history /home/ssm-user/.bash_history /home/ubuntu/.bash_history /root/.bash_history && sudo apt clean && sudo rm -rf /tmp/* /var/tmp/*
sudo journalctl --vacuum-time=1s && sudo find /var/log -type f -name "*.gz" -delete && sudo find /var/log -type f -exec truncate -s 0 {} \;

If the raw image was taken before these fixes, apply them now: the setuptools fix (phase 14), the MariaDB strict-mode lines (phase 13), certbot (phase 12) and the nginx log format file (phase 20).

sudo cloud-init clean --logs && sudo truncate -s 0 /etc/machine-id && echo "SANITISED - stop this instance from the console"

17.3 Create the golden AMI

AWS console Stop the builder from the console. Don't reboot it from the terminal: a reboot would undo the clean-up. Then create the image Canofy-Frappe-v15-Golden-<YYYYMMDD> with tags Type=GoldenAMI, FrappeVersion, CanofyTCS, BaseOS=Ubuntu22.04.

17.4 Clean up

Terminate the builder, and deregister the raw AMI and delete its snapshot. The raw image contains the deploy keys, so never share it.

When using the golden AMI for a new customer: set the hostname, change the MariaDB root password, create new deploy keys and pull the latest code, then continue from phase 18.

18. Create the site and install the apps

Server via SSM, as frappe

cd ~/frappe-bench && bench config dns_multitenant on
bench new-site bel.canofy.tech

Enter the MariaDB root password, then a new Administrator password (save it in the password manager).

bench --site bel.canofy.tech install-app s3_file_storage
bench --site bel.canofy.tech install-app canofy_tcs
Figure 24 canofy_tcs installed: DocTypes 100%, Address customisations, Dashboard.

18.1 Customer app

Download it with the same token method as phase 16 (the Belimo repo is under the canofytech organisation):

GIT_TERMINAL_PROMPT=0 git ls-remote --symref https://github.com/canofytech/canofy_custom_BEL.git HEAD
bench get-app https://github.com/canofytech/canofy_custom_BEL.git --branch main && ls apps/

The Python app name can differ from the repo name (Belimo: custom_belimo). Frappe 15.72.5 resolves required_apps on GitHub under the frappe/erpnext organisations, and fails for private dependencies. So comment the line out temporarily, install the app, and then restore the file:

grep -n "required_apps" apps/custom_belimo/custom_belimo/hooks.py
sed -i 's/^required_apps/# required_apps/' apps/custom_belimo/custom_belimo/hooks.py && bench --site bel.canofy.tech install-app custom_belimo
git -C apps/custom_belimo checkout -- custom_belimo/hooks.py && git -C apps/custom_belimo status --short && echo "hooks restored"

18.2 Site settings and checks

bench --site bel.canofy.tech set-config host_name "https://bel.canofy.tech" && bench --site bel.canofy.tech enable-scheduler && bench --site bel.canofy.tech set-maintenance-mode off
bench --site bel.canofy.tech set-config -p server_script_enabled 1
bench --site bel.canofy.tech list-apps
Figure 25 SS-22: apps on the BEL site: frappe 15.72.5 (pinned tag), canofy_tcs 2.3.0 development, s3_file_storage 0.0.1, custom_belimo 1.2.3. canofy_ddit stays in the bench only.

bench doctor reports Connection refused 127.0.0.1:11000 until production mode is set up. That's expected at this point.

19. Production mode (nginx and supervisor)

Server via SSM, as frappe Size the web workers for the instance. Use 3 on t3a.medium:

bench config set-common-config -c gunicorn_workers 3
sudo bench setup production frappe --yes

Check that all three parts worked. In the Belimo run, setup production stopped silently before linking supervisor (because certbot was missing).

ls -la /etc/supervisor/conf.d/ /etc/nginx/conf.d/ && sudo ls /etc/sudoers.d/

You should see frappe-bench.conf in both folders, and a frappe file in sudoers.d. If the supervisor link is missing:

sudo ln -sf /home/frappe/frappe-bench/config/supervisor.conf /etc/supervisor/conf.d/frappe-bench.conf && grep -q chown /etc/supervisor/supervisord.conf || sudo sed -i '/^\[unix_http_server\]/a chown=frappe:frappe' /etc/supervisor/supervisord.conf
sudo systemctl restart supervisor && sleep 8 && sudo supervisorctl status

Remove nginx's default welcome page:

sudo rm -f /etc/nginx/sites-enabled/default && sudo nginx -t && sudo systemctl reload nginx
curl -sI -H "Host: bel.canofy.tech" http://127.0.0.1 | head -3 && bench doctor
Figure 26 SS-23: sudoers set up, setup production complete, all 7 processes RUNNING, HTTP 200.

20. SSL (Let's Encrypt)

Server via SSM, as ssm-user DNS must resolve first (phase 10). Define the main log format before enabling SSL. Bench's SSL nginx template uses it, and Ubuntu doesn't define it:

echo "log_format main '\$remote_addr - \$remote_user [\$time_local] \"\$request\" \$status \$body_bytes_sent \"\$http_referer\" \"\$http_user_agent\" \"\$http_x_forwarded_for\"';" | sudo tee /etc/nginx/conf.d/00-log-format.conf
sudo certbot certonly --nginx -d bel.canofy.tech -m devops@canofy.tech --agree-tos --no-eff-email --deploy-hook "systemctl reload nginx"
sudo -iu frappe bash -c 'cd ~/frappe-bench && bench --site bel.canofy.tech set-config ssl_certificate /etc/letsencrypt/live/bel.canofy.tech/fullchain.pem && bench --site bel.canofy.tech set-config ssl_certificate_key /etc/letsencrypt/live/bel.canofy.tech/privkey.pem && bench setup nginx --yes'
sudo nginx -t && sudo systemctl reload nginx && curl -sI https://bel.canofy.tech | head -1 && curl -sI http://bel.canofy.tech | head -1 && sudo certbot renew --dry-run

Expected: 200 over https, 301 over http, and all simulated renewals succeeded.

Figure 27 SS-25: log format file, nginx test OK, HTTPS 200, HTTP 301, renewal dry-run succeeded.
Figure 28 SS-24: login page served at https://bel.canofy.tech, before branding.

21. Daily database backups to S3

Server via SSM, as frappe The script is written with a single line:

printf '%s\n' '#!/bin/bash' 'set -e' 'export PATH=/usr/local/bin:/usr/bin:/bin:/snap/bin' 'cd /home/frappe/frappe-bench' 'echo "=== $(date) start"' 'bench --site bel.canofy.tech backup --compress' 'aws s3 sync sites/bel.canofy.tech/private/backups/ s3://bel.canofy.tech/_backups/ --only-show-errors --region ap-south-1' 'echo "=== $(date) done"' > ~/backup_to_s3.sh && chmod 700 ~/backup_to_s3.sh
~/backup_to_s3.sh && aws s3 ls s3://bel.canofy.tech/_backups/ --region ap-south-1 --human-readable
(crontab -l 2>/dev/null; echo '30 1 * * * /home/frappe/backup_to_s3.sh >> /home/frappe/frappe-bench/logs/backup_to_s3.log 2>&1') | crontab - && crontab -l

Then create lifecycle Rule 2 for _backups/ (phase 7.3).

Figure 29 SS-26: scheduler enabled; bench's 6-hourly local backup plus the 01:30 S3 job; test backup (330.2 KiB) in _backups/.

22. Canofy configuration (Setup Sheet 1.1–2.2, scripted)

The script canofy_site_setup.py (appendix A) applies Website Settings, System Settings, Navbar, Core Apps, Attachments and S3 File Settings from a per-customer JSON file (appendix B). It runs everything in one transaction, rolls back on any error, and is safe to re-run.

22.1 Prepare the files

  1. Copy bel_setup.json to <code>_setup.json and change: customer_code, the s3_settings bucket and folder, core_apps.apps (the customer app), and anything else customer-specific.
  2. Keep "enabled": false on the connected_app and email_account sections unless you set up email through the script (phase 24 uses the manual path).
  3. AWS console Upload to s3://bel.canofy.tech/_setup/: the .py, the .json, and a folder branding/ containing Logo 150x150.png, Canofy logo 150px.png and Canofy Favicon1.png (names must match exactly).

Warning: browsers rename repeated downloads (for example canofy_site_setup (1).py). Upload the newest file, and check the version on the server with grep.

22.2 Download and dry-run

Server via SSM, as frappe

mkdir -p ~/canofy_setup && aws s3 cp s3://bel.canofy.tech/_setup/ ~/canofy_setup/ --recursive --region ap-south-1 && ls -R ~/canofy_setup

If the logos landed next to the script instead of in branding/:

mkdir -p ~/canofy_setup/branding && mv ~/canofy_setup/*.png ~/canofy_setup/branding/
cd ~/frappe-bench/sites && ../env/bin/python ~/canofy_setup/canofy_site_setup.py apply --site bel.canofy.tech --config ~/canofy_setup/bel_setup.json --dry-run

The dry run ends with DRY RUN finished - nothing was saved. If it reports unknown field(s) or Could not find, correct the JSON (the valid fields are listed in the error) and run it again.

22.3 Apply

../env/bin/python ~/canofy_setup/canofy_site_setup.py apply --site bel.canofy.tech --config ~/canofy_setup/bel_setup.json
Figure 30 SS-AUTO-01: Belimo apply output: logos placed as local public files, settings changed, help menu, Core Apps, 5 Attachments, S3 File Settings, "APPLIED and committed".

Run the same command once more. Every section should report "already up to date", "already present" or "exists, skipped".

Canofy in the browser Hard-refresh with Ctrl+Shift+R and check: the Canofy logo on the login page and navbar; the favicon; Help → About Canofy; no "Login with Email Link". Then attach a test file to any record, and check that it appears in S3:

Server via SSM, as frappe

aws s3 ls s3://bel.canofy.tech/BEL/ --recursive --region ap-south-1 | tail -3
Screenshot to capture, SS-27: Login page and desk with Canofy branding

23. Users and passwords

Server via SSM, as frappe Set or rotate the Administrator password. The command prompts for it, so it isn't saved in shell history.

bench --site bel.canofy.tech set-admin-password --logout-all-sessions
bench --site bel.canofy.tech set-password no-reply-bel@canofy.tech --logout-all-sessions

Customer users, roles and the Business Partner master are loaded from the customer's data (Setup Sheet 5–7).

24. Microsoft 365 email (ICEGATE mailbox)

This follows the existing Canofy pattern (Canofy-COS, -BOS, -EAP, -LAX, -SLC): one Entra app per customer, Microsoft Graph delegated permissions, and a Frappe Connected App plus an Email Account. Incoming mail lands in Icegate Intimation.

24.1 Mailbox

The mailbox no-reply-<code>@canofy.tech is created by the M365 admin, with display name Canofy Support [<Customer>] and IMAP enabled. Authenticated SMTP must also be enabled if Canofy sends mail from it.

24.2 Entra app registration

Microsoft Entra Sign in at entra.microsoft.com as the mailbox user (or an admin). Belimo's tenant allows normal users to register apps.

Figure 31 Signed in as no-reply-bel: Canofy Technologies tenant, display name "Canofy Support [Belimo]", no roles needed.
  1. App registrations → New registration: name Canofy-BEL, account type Single tenant, redirect URI Web. You'll replace it in 24.4 with the value Frappe generates, so any placeholder will do for now.
  2. Note the Application (client) ID and the Directory (tenant) ID.
Figure 32 SS-34: Canofy-BEL overview (IDs masked): single tenant, 1 web redirect URI, Activated.
  1. API permissions → Add a permission → Microsoft Graph → Delegated: add IMAP.AccessAsUser.All, SMTP.Send and offline_access (User.Read is usually there already). "Grant admin consent" isn't needed, because the user consents when signing in.
Figure 33 SS-35: Microsoft Graph (4), all delegated, admin consent not required.
  1. Certificates & secrets → New client secret: description Canofy-BEL-secret, expiry per policy. Copy the Value immediately into the password manager, together with the expiry date.
Figure 34 SS-36: Canofy-BEL-secret, expires 25 Sep 2027 (value and secret ID masked).
  1. Overview → Endpoints: copy OpenID Connect metadata document https://login.microsoftonline.com/<tenant-id>/v2.0/.well-known/openid-configuration.

Note: if the app was registered by the mailbox account, ask an admin to add themselves under Owners. Put the secret expiry in the calendar, about two weeks early. When the secret expires, mail pull stops without warning.

24.3 Canofy user for the mailbox

Canofy in the browser Logged in as Administrator, go to User → Add: email no-reply-bel@canofy.tech, first name Canofy Support, last name [Belimo], Send Welcome Email unticked. Save, add the role System Manager, and set a strong Canofy password (phase 23). This is the connected user: the Microsoft token is stored under this Canofy user.

24.4 Connected App

Canofy in the browser Connected App → Add:

Field Value
Provider Name Microsoft 365
OpenID Configuration the URL from 24.2, then click Get OpenID Configuration
Client ID the Canofy-BEL client ID
Client Secret the secret value
Scopes https://outlook.office.com/IMAP.AccessAsUser.All, https://outlook.office.com/SMTP.Send, offline_access

Save, then copy the generated Redirect URI (https://bel.canofy.tech/api/method/frappe.integrations.doctype.connected_app.connected_app.callback/<name>).

Microsoft Entra Canofy-BEL → Authentication → Web → Redirect URIs: replace the placeholder with that exact URI, then Save.

Screenshot to capture, SS-37: Saved Connected App, and the Entra Authentication page with the matching redirect URI

24.5 Email Account

Canofy in the browser Email Account → Add. Leave incoming and outgoing off for now, because saving with them on tests the connection, which fails before authorization.

Field Value
Email Address / Account Name no-reply-bel@canofy.tech / No Reply Bel
Authentication Method OAuth; Connected App from 24.4; Connected User no-reply-bel@canofy.tech
Incoming IMAP, SSL, outlook.office365.com, port 993, sync UNSEEN, initial sync 250, attachment limit 50, create contacts on
IMAP Folder Folder INBOX → Append To Icegate Intimation (this is the ICEGATE filtering)
Outgoing smtp.office365.com, port 587, TLS; always use the account email address and name as sender; unsubscribe message on; track status on

24.6 Authorize, enable and test

  1. Canofy in the browser In a private window, log in to Canofy as no-reply-bel@canofy.tech, open the Email Account, and click Authorize API Access. Sign in to Microsoft as the mailbox and Accept.
  2. Tick Enable Incoming, Default Incoming, Enable Outgoing and Default Outgoing, then Save. The save succeeding means the connection test passed.
  3. Send a test email to the mailbox. Go to Scheduled Job Type → email_account.pull → Execute, then check Icegate Intimation and Communication for the new record.
Screenshot to capture, SS-38: Icegate Intimation record created from the test email

Belimo: email was activated. The end-to-end ICEGATE test still has to be recorded.

25. OCR service

The shared OCR server (172.31.35.206:5000, POST /generate_ocr, gunicorn service ocr_app) identifies each ERP server by its private IP in property.json. It checks the username and password the ERP server sends (stored Fernet-encrypted with the ENCRYPTION_KEY in .env), then reads PDFs from that customer's bucket.

25.1 Network

AWS console OCR server's security group → Inbound → add TCP 5000 from 172.31.10.27/32, description BEL bel.canofy.tech.

Server via SSM, as ssm-user Test from the customer server. Any HTTP code means the server is reachable; 000 means it's blocked.

curl -s -m 5 -o /dev/null -w "%{http_code}\n" -X POST http://172.31.35.206:5000/generate_ocr -H "Content-Type: application/json" -d "{}"

25.2 Find the live property.json

OCR server, as ubuntu

systemctl cat ocr_app.service | grep -E "WorkingDirectory|ExecStart"
Figure 35 The service runs gunicorn from /home/ubuntu/canofy_OCR, so the live file is /home/ubuntu/canofy_OCR/property.json.
Figure 36 Several copies exist (backups and a stray /home/property.json). Edit only the copy in the service's working directory.

Work as ubuntu. As ssm-user, you get "Permission denied" on /home/ubuntu.

sudo -iu ubuntu
Figure 37 ssm-user can't list /home/ubuntu; switch with sudo -iu ubuntu first.

25.3 Add the customer entry

OCR server, as ubuntu As ubuntu, back up the file first:

cp ~/canofy_OCR/property.json ~/canofy_OCR/property.json.bak.$(date +%Y%m%d-%H%M) && ls -la ~/canofy_OCR/property.json*

Generate credentials, encrypted with the existing key. Never change ENCRYPTION_KEY: that would break every customer. This prints the two Fernet tokens and the three bench commands for the customer server. Don't share the output.

cd ~/canofy_OCR && /usr/bin/python3 -c "import secrets,base64;from cryptography.fernet import Fernet;k=[l.split('=',1)[1].strip().strip('\"') for l in open('.env') if l.startswith('ENCRYPTION_KEY=')][0];f=Fernet(k.encode());u='bel_ocr_'+secrets.token_hex(3);p=secrets.token_urlsafe(24);print('ENC_USERNAME =',f.encrypt(u.encode()).decode());print('ENC_PASSWORD =',f.encrypt(p.encode()).decode());print(' bench --site bel.canofy.tech set-config ocr_username',base64.b64encode(u.encode()).decode());print(' bench --site bel.canofy.tech set-config ocr_password',base64.b64encode(p.encode()).decode());print(' bench --site bel.canofy.tech set-config ocr_url http://172.31.35.206:5000/generate_ocr')"

Edit the file with nano ~/canofy_OCR/property.json (paste with right-click, save with Ctrl+O and Enter, exit with Ctrl+X). Add a comma after the last entry, then:

"172.31.10.27": {
    "BUCKET NAME": "bel.canofy.tech",
    "username": "PASTE_ENC_USERNAME",
    "password": "PASTE_ENC_PASSWORD"
}

Validate the file (this prints only IPs, buckets and True/False):

cd ~/canofy_OCR && /usr/bin/python3 -c "import json;from cryptography.fernet import Fernet;k=[l.split('=',1)[1].strip().strip('\"') for l in open('.env') if l.startswith('ENCRYPTION_KEY=')][0];f=Fernet(k.encode());d=json.load(open('property.json'));[print(ip,'->',v.get('BUCKET NAME'),'| decrypts:',all(bool(f.decrypt(str(v[x]).strip('b').strip(\"'\").encode())) for x in ('username','password'))) for ip,v in d.items()]"

If it fails, restore the backup with cp "$(ls -t ~/canofy_OCR/property.json.bak.* | head -1)" ~/canofy_OCR/property.json.

Alternative: ocr_add_customer.py (appendix D) does all of 25.3 with a dry run and a protection against overwriting an existing entry.

25.4 Customer site config and restart

Server via SSM, as frappe In ~/frappe-bench, paste the three bench … set-config lines (each starts with a space). Then check that the keys exist, without showing their values:

grep -o '"ocr_[a-z]*"' sites/bel.canofy.tech/site_config.json && bench restart

To build the base64 values by hand, use printf '%s' 'value' | base64. echo adds a newline that breaks the match.

OCR server, as ubuntu Restart the OCR service at a quiet moment (it's shared with all customers):

sudo systemctl restart ocr_app.service && sudo systemctl status ocr_app.service --no-pager | head -5

25.5 End-to-end test

Canofy in the browser Run OCR on a real document in the customer site, the way users do. Then check the OCR server log:

OCR server, as ubuntu

sudo journalctl -u ocr_app.service --since "15 minutes ago" --no-pager | tail -80

Belimo (open): BEL reaches OCR, but OCR returns 500 INTERNAL SERVER ERROR (boe_bridge.request_ocr_data). Use the checklist in section 28 to find the cause.

26. Go-live checklist and clean-up

# Item Belimo
1 Delete /etc/sudoers.d/90-frappe-setup (sudo rm then sudo visudo -c) Open
2 Deploy keys added by an admin; remotes switched; temporary GitHub tokens deleted Open
3 Disk expanded to 40 GB (EC2 Modify volume, then sudo growpart /dev/nvme0n1 1 && sudo resize2fs /dev/nvme0n1p1) Open
4 IMDSv2 set to required Open (deviation)
5 Session Manager logging enabled Open
6 Administrator password rotated and strong; real Administrator email set before 2FA Open
7 2FA enabled (System Settings, method Email) after email works Open
8 ICEGATE email end-to-end test recorded Open
9 OCR end-to-end test passes Failing (HTTP 500)
10 First DLM snapshot visible To check
11 _backups/ lifecycle rule created; bucket tag Customer fixed To confirm
12 _setup/ deleted from the bucket Open
13 Golden AMI parts 17.2–17.4 completed; raw image deleted Open
14 Customer handover email sent (section 27) Draft ready

27. Customer handover email

Send this once UAT can start. Credentials are always sent separately, never in this email.

Subject: Canofy Production Environment Ready – <Customer Legal Name>

Dear <Customer Contact Name>,

We are pleased to inform you that the dedicated Canofy production environment for <Customer Legal Name> has been set up and is ready for your team.

Access details
- Application URL: https://<code>.canofy.tech
- Login credentials will be shared separately with each nominated user. For security reasons, we never send passwords by email.

What has been configured
- A dedicated, secured production instance with HTTPS encryption
- Secure cloud storage for all document attachments
- A dedicated mailbox for ICEGATE intimation emails: no-reply-<code>@canofy.tech
- Automated OCR processing of customs documents
- Automated daily backups

Security policies in place
- Automatic logout after 30 minutes of inactivity
- One active session per user
- The account is temporarily locked after 3 consecutive failed login attempts
- Two-factor authentication (email OTP) will be enabled before go-live

Next steps
1. Please share the names, email IDs and roles of the users who need access.
2. Please share your Business Partner master and item details.
3. Please make sure ICEGATE intimation emails are routed to no-reply-<code>@canofy.tech.
4. UAT window: <start> to <end>.
5. Go-live: <date>, after UAT sign-off.

Warm regards,
<Name>, <Designation>
Canofy Technologies Private Limited

28. Troubleshooting

Every issue below was hit during the Belimo onboarding. The steps above already include the fixes.

Symptom Cause Fix
MariaDB won't start after writing the config The SSM terminal merged heredoc lines (256M[mysql]) Phase 13 printf command; check with cat -A
bench init asks for a sudo password, then fails at sudo supervisorctl status frappe has no password 90-frappe-setup NOPASSWD (phase 12); answer y to the rollback, then rerun
could not read Username / Password authentication is not supported Private repo; a password was entered instead of a token, or the paste failed Phase 16.1 token method (check the printed length: 40)
Fine-grained token can't see the repos Token owned by your personal account, not the organisation Use a classic token with repo scope, 7 days
new-site: No module named 'pkg_resources' The uv-built environment lacks setuptools Phase 14 setuptools fix; bench drop-site <site> --force --no-backup, then recreate
Row size too large (> 8126) on tabBOE Items MariaDB strict mode innodb_strict_mode = OFF (phase 13); then install-app
InvalidRemoteException (GitHub 404 on erpnext/canofy_tcs) required_apps lookup in Frappe 15.72.5 Phase 18.1 comment-out workaround
bench doctor: Connection refused 127.0.0.1:11000 Production mode not set up yet Phase 19
502 Bad Gateway; supervisorctl status is empty setup production stopped at the sudoers step (no certbot), so supervisor wasn't linked Install certbot; phase 19 manual link and chown
Certbot is not installed during setup sudoers certbot missing Phase 12 certbot step
unknown log format "main" Ubuntu nginx doesn't define it Phase 20 00-log-format.conf
Setup script: LinkValidationError (FTWZ BOE / DTA) Sheet labels used instead of DocType names Use FTWZ BOE Import and FTWZ DTA Sale
Setup script: TypeError ... got 'NoneType' in s3_file_storage The S3 hook ran on logo upload before the S3 settings existed Use the current script, which places logos without hooks
grep -c placed returns 0 An old copy of the script was uploaded Upload the newest download (watch for "(1)")
set-config -p ... true raises malformed node -p expects a Python literal Use 1 or 0
OCR returns 500 Under investigation for BEL See the checklist below

OCR 500 checklist. Compare the service start time with the property.json change time, and restart if the file is newer (systemctl show ocr_app.service -p ActiveEnterTimestamp). Check that the new entry decrypts (phase 25.3 validation). Check that the base64-decoded ocr_username on the customer server matches the OCR username exactly. Check that the PDF exists under the S3 path the OCR log mentions (aws s3 ls s3://bel.canofy.tech/BEL/ --recursive), and that the folder case matches (MIL uses lower-case mil). Read the traceback in journalctl -u ocr_app.service.

Error screenshots

Figure 38 MariaDB failed: heredoc lines merged by the SSM terminal.
Figure 39 bench init stuck at a sudo password prompt for frappe.
Figure 40 bench init traceback at sudo supervisorctl status, with the rollback prompt.
Figure 41 The bench folder wasn't found on another server: always locate it first with find … -name apps.txt.
Figure 42 Loop variables are lost inside sudo -iu … bash -c '…': switch user first.
Figure 43 bench get-app failed authentication: a password or failed paste instead of a token.
Figure 44 Expected "no such group" warning after get-app, before production mode.
Figure 45 new-site failed: No module named pkg_resources (Dropbox Settings import).
Figure 46 install-app failed: Row size too large on tabBOE Items.
Figure 47 install-app custom_belimo: InvalidRemoteException.
Figure 48 Traceback showing the required_apps lookup on GitHub (org erpnext → 404).
Figure 49 bench doctor: Redis 11000 refused before production mode.
Figure 50 The same Redis error after a setup production that didn't complete.
Figure 51 502 Bad Gateway: nginx linked, but /etc/supervisor/conf.d is empty.
Figure 52 supervisorctl status shows no processes.
Figure 53 After the manual supervisor link and chown: 7 processes RUNNING, HTTP 200.
Figure 54 setup sudoers failed: certbot not installed.
Figure 55 Certificate issued, but nginx -t fails with unknown log format "main".
Figure 56 Only the logos reached S3, in the wrong folder, so the script was missing.
Figure 57 Dry run: System Settings, Navbar and Core Apps OK; Attachments failed on FTWZ names (rolled back).
Figure 58 Apply failed in the s3_file_storage hook before the S3 settings existed (rolled back).
Figure 59 grep -c placed = 0: an old script was still in S3.
Figure 60 The OCR policy editor with BEL lines added. Without Next → Save changes, this isn't saved.
Figure 61 set-config -p server_script_enabled true raises a malformed node error; use 1.

29. Maintenance

Task When How
Entra client secret renewal Before expiry (BEL: 25 Sep 2027) New secret → Connected App → Client Secret → Save; no re-authorization needed
SSL certificate Automatic (certbot timer) Check with sudo certbot renew --dry-run
Backups Weekly aws s3 ls s3://bel.canofy.tech/_backups/; check logs/backup_to_s3.log
Disk Monthly df -h /; expand before 80%
Frappe / app updates Planned change Update the version lock, bench update on a test copy first; rebuild the golden AMI

Appendices

The scripts referenced in this SOP, as used for Belimo. Keep the master copies in the Canofy DevOps repository.

A. canofy_site_setup.py

Phase 22 (and optional email sections). Run from ~/frappe-bench/sites with ../env/bin/python.

#!/usr/bin/env python3
"""
Canofy site setup automation  (Setup Sheet 1.1, 1.2, 1.3, 2, 2.2)

Run as the frappe user, FROM THE sites FOLDER, with the bench's Python:

  cd ~/frappe-bench/sites
  ../env/bin/python ~/canofy_setup/canofy_site_setup.py inspect --site bel.canofy.tech
  ../env/bin/python ~/canofy_setup/canofy_site_setup.py apply   --site bel.canofy.tech --config ~/canofy_setup/bel_setup.json --dry-run
  ../env/bin/python ~/canofy_setup/canofy_site_setup.py apply   --site bel.canofy.tech --config ~/canofy_setup/bel_setup.json

Modes
  inspect   Print the real field names of the doctypes this script touches
            (and any extra ones given with --doctype). Changes nothing.
  apply     Apply the JSON config. Safe to re-run: unchanged values are left
            alone, existing records are skipped, logos are reused.
            With --dry-run everything is checked and printed, then rolled back.

Config conventions
  "@<filename>"   -> upload <branding_dir>/<filename> as a PUBLIC file and use its URL
  keys starting with "_" are comments and are ignored
  a section with "enabled": false is skipped
"""
import argparse
import json
import os
import sys

LAYOUT_TYPES = {"Section Break", "Column Break", "Tab Break", "HTML", "Button", "Fold", "Heading"}
DEFAULT_INSPECT = ["Website Settings", "System Settings", "Navbar Settings", "Navbar Item", "Core Apps", "Attachments", "S3 File Settings", "Connected App", "Email Account"]


def log(msg=""):
    print(msg, flush=True)


def clean(d):
    """Drop comment keys (starting with '_') and the 'enabled' switch."""
    return {k: v for k, v in d.items() if not k.startswith("_") and k != "enabled"}


class Ctx:
    def __init__(self, frappe, cfg, dry_run, new_secret=False):
        self.frappe = frappe
        self.cfg = cfg
        self.dry_run = dry_run
        self.new_secret = new_secret
        self.branding_dir = os.path.expanduser(cfg.get("branding_dir", "~/canofy_setup/branding"))
        self.file_cache = {}

    # ---------- files ----------
    def resolve(self, value):
        if isinstance(value, str) and value.startswith("@"):
            return self.upload_public_file(os.path.join(self.branding_dir, value[1:]))
        return value

    def upload_public_file(self, path):
        frappe = self.frappe
        if path in self.file_cache:
            return self.file_cache[path]
        if not os.path.isfile(path):
            raise SystemExit(f"ERROR: logo file not found: {path}")
        fname = os.path.basename(path)
        existing = frappe.get_all(
            "File",
            filters={"file_name": fname, "is_private": 0, "attached_to_doctype": ["is", "not set"]},
            fields=["name", "file_url"],
            limit=1,
        )
        if existing:
            url = existing[0].file_url
            log(f"  = file '{fname}' already uploaded -> {url}")
        elif self.dry_run:
            url = f"/files/{fname}"
            log(f"  + would upload '{fname}' (public) -> {url}")
        else:
            # Place the file the way Frappe stores public files, and register it
            # WITHOUT running File hooks. The s3_file_storage app hooks every File
            # insert and would try to push branding to S3 (and fails before S3
            # settings exist). Branding must stay local + public for the login page.
            import hashlib
            import shutil
            from frappe.utils import get_files_path

            with open(path, "rb") as f:
                content = f.read()
            dest = get_files_path(fname)  # sites/<site>/public/files/<fname>
            if not os.path.exists(dest):
                shutil.copyfile(path, dest)
            doc = frappe.new_doc("File")
            doc.update({
                "file_name": fname,
                "file_url": f"/files/{fname}",
                "is_private": 0,
                "folder": "Home",
                "file_size": len(content),
                "content_hash": hashlib.md5(content).hexdigest(),
            })
            doc.set_new_name()
            doc.set_user_and_timestamp()
            doc.db_insert()
            url = doc.file_url
            log(f"  + placed '{fname}' as local public file -> {url}")
        self.file_cache[path] = url
        return url


# ---------- helpers ----------
def require_doctype(frappe, dt):
    if not frappe.db.exists("DocType", dt):
        raise SystemExit(f"ERROR: DocType '{dt}' does not exist on this site. Run 'inspect' to find the right name.")
    return frappe.get_meta(dt)


def check_fields(meta, keys, where):
    bad = [k for k in keys if not meta.has_field(k)]
    if bad:
        valid = ", ".join(df.fieldname for df in meta.fields if df.fieldtype not in LAYOUT_TYPES)
        raise SystemExit(f"ERROR: {where}: unknown field(s) {bad}.\n  Valid fields: {valid}")


def same(a, b):
    norm = lambda x: "" if x is None else str(x)
    return norm(a) == norm(b)


# ---------- sections ----------
def set_single(ctx, dt, values):
    frappe = ctx.frappe
    values = clean(values)
    meta = require_doctype(frappe, dt)
    check_fields(meta, values.keys(), dt)
    doc = frappe.get_single(dt)
    changed = []
    for key, raw in values.items():
        new = ctx.resolve(raw)
        old = doc.get(key)
        if not same(old, new):
            doc.set(key, new)
            changed.append((key, old, new))
    if not changed:
        log(f"  {dt}: already up to date")
        return
    for key, old, new in changed:
        log(f"  {dt}.{key}: {old!r} -> {new!r}")
    doc.save(ignore_permissions=True)


def navbar_help(ctx, cfg):
    frappe = ctx.frappe
    item_meta = require_doctype(frappe, "Navbar Item")
    doc = frappe.get_single("Navbar Settings")
    rows = {row.item_label: row for row in doc.get("help_dropdown") or []}
    changed = False

    for label in cfg.get("hide", []):
        row = rows.get(label)
        if not row:
            log(f"  help item '{label}' not found (skipped)")
        elif not row.hidden:
            row.hidden = 1
            changed = True
            log(f"  hide help item '{label}'")
        else:
            log(f"  help item '{label}' already hidden")

    for item in cfg.get("add", []):
        item = clean(item)
        check_fields(item_meta, item.keys(), "Navbar Item")
        row = rows.get(item["item_label"])
        if row:
            diffs = {k: v for k, v in item.items() if not same(row.get(k), v)}
            if diffs:
                row.update(diffs)
                changed = True
                log(f"  update help item '{item['item_label']}': {diffs}")
            else:
                log(f"  help item '{item['item_label']}' already present")
        else:
            doc.append("help_dropdown", item)
            changed = True
            log(f"  add help item '{item['item_label']}'")

    if changed:
        doc.save(ignore_permissions=True)
    else:
        log("  Navbar help menu: already up to date")


def core_apps(ctx, cfg):
    """Fill the Core Apps table from the apps installed on this bench.
    Title, version and git branch are read from the bench itself, so re-running
    this after an upgrade refreshes the shown versions."""
    frappe = ctx.frappe
    from frappe.utils.change_log import get_versions

    dt = cfg.get("doctype", "Core Apps")
    table = cfg.get("table_field", "apps")
    meta = require_doctype(frappe, dt)
    check_fields(meta, [table], dt)
    child_meta = frappe.get_meta(meta.get_field(table).options)

    installed = frappe.get_installed_apps()
    versions = get_versions()
    wanted = []
    for app in cfg["apps"]:
        if app not in installed:
            raise SystemExit(f"ERROR: Core Apps: '{app}' is not installed on this site")
        v = versions.get(app, {})
        row = {
            "app_name": app,
            "title": v.get("title") or app,
            "app_version": v.get("version") or "",
            "git_branch": v.get("branch") or "",
        }
        wanted.append({k: val for k, val in row.items() if child_meta.has_field(k)})

    doc = frappe.get_single(dt)
    current = doc.get(table) or []
    up_to_date = len(current) == len(wanted) and all(
        all(same(r.get(k), w[k]) for k in w) for r, w in zip(current, wanted)
    )
    if up_to_date:
        log(f"  {dt}: already up to date")
        return
    doc.set(table, [])
    for w in wanted:
        doc.append(table, w)
        log(f"  {dt} row: {w}")
    doc.save(ignore_permissions=True)


def connected_app(ctx, cfg):
    """Create/update a Connected App with a FIXED name, so its redirect URI is
    known in advance and matches the one registered in Entra. The client secret
    is asked at a hidden prompt (never stored in the JSON)."""
    import getpass

    frappe = ctx.frappe
    cfg = clean(cfg)
    name = cfg.pop("name")
    scopes = cfg.pop("scopes", [])
    meta = require_doctype(frappe, "Connected App")
    check_fields(meta, list(cfg.keys()) + ["scopes"], "Connected App")
    scope_meta = frappe.get_meta(meta.get_field("scopes").options)
    check_fields(scope_meta, ["scope"], meta.get_field("scopes").options)

    # Fill the endpoint fields from the OpenID configuration document
    endpoints = {}
    if cfg.get("openid_configuration"):
        import requests

        data = requests.get(cfg["openid_configuration"], timeout=20).json()
        mapping = {
            "authorization_uri": "authorization_endpoint",
            "token_uri": "token_endpoint",
            "revocation_uri": "revocation_endpoint",
            "userinfo_uri": "userinfo_endpoint",
            "introspection_uri": "introspection_endpoint",
        }
        for field, key in mapping.items():
            if meta.has_field(field) and data.get(key):
                endpoints[field] = data[key]

    exists = frappe.db.exists("Connected App", name)
    doc = frappe.get_doc("Connected App", name) if exists else frappe.new_doc("Connected App")
    changed = []
    for key, val in {**cfg, **endpoints}.items():
        if not same(doc.get(key), val):
            doc.set(key, val)
            changed.append(key)
    if [r.scope for r in doc.get("scopes") or []] != scopes:
        doc.set("scopes", [{"scope": s} for s in scopes])
        changed.append("scopes")

    if not exists or ctx.new_secret:
        if ctx.dry_run:
            log("  would prompt for the client secret (hidden)")
        else:
            secret = getpass.getpass("  Paste the Entra client secret VALUE (hidden), then Enter: ").strip()
            if not secret:
                raise SystemExit("ERROR: empty client secret")
            doc.client_secret = secret
            changed.append("client_secret")

    if not changed:
        log(f"  Connected App '{name}': already up to date")
    else:
        log(f"  Connected App '{name}': {'update' if exists else 'create'} -> {', '.join(changed)}")
        if exists:
            doc.save(ignore_permissions=True)
        else:
            try:
                doc.insert(ignore_permissions=True, set_name=name)
            except TypeError:
                doc.name = name
                doc.flags.name_set = True
                doc.insert(ignore_permissions=True)
    log(f"  scopes      : {scopes}")
    log(f"  redirect URI: {doc.get('redirect_uri')}")
    if not ctx.dry_run and doc.name != name:
        log(f"  WARNING: Frappe named it '{doc.name}', not '{name}'. Put the redirect URI above into Entra.")


def email_account(ctx, cfg):
    """Create/update an OAuth Email Account (mirrors MIL). Incoming/outgoing are
    switched on only once a Microsoft token exists for connected_app + user,
    i.e. after 'Authorize API Access' has been done once in the browser."""
    frappe = ctx.frappe
    cfg = clean(cfg)
    later = cfg.pop("enable_after_authorize", {})
    meta = require_doctype(frappe, "Email Account")
    check_fields(meta, list(cfg.keys()) + list(later.keys()), "Email Account")
    name = cfg["email_account_name"]
    ca, user = cfg.get("connected_app"), cfg.get("connected_user")

    authorized = False
    try:
        authorized = bool(ca and user and frappe.db.exists("Token Cache", {"connected_app": ca, "user": user}))
    except Exception as e:  # field names differ on some versions
        log(f"  (could not check Token Cache: {e})")

    values = dict(cfg)
    if authorized:
        values.update(later)

    exists = frappe.db.exists("Email Account", name)
    doc = frappe.get_doc("Email Account", name) if exists else frappe.new_doc("Email Account")
    changed = []
    for key, val in values.items():
        if isinstance(val, list):  # child table, e.g. imap_folder
            cur = [{k: r.get(k) for k in (val[0].keys() if val else [])} for r in doc.get(key) or []]
            if cur != val:
                doc.set(key, val)
                changed.append(key)
        elif not same(doc.get(key), val):
            doc.set(key, val)
            changed.append(key)

    if not changed:
        log(f"  Email Account '{name}': already up to date")
    else:
        log(f"  Email Account '{name}': {'update' if exists else 'create'} -> {', '.join(changed)}")
        if exists:
            doc.save(ignore_permissions=True)
        else:
            doc.insert(ignore_permissions=True)

    if authorized:
        log(f"  Microsoft token found for {user}: incoming/outgoing settings applied.")
    else:
        log("  NEXT (manual, once): log in to the site as " + str(user) + ", open Email Account '" + name + "',")
        log("  click 'Authorize API Access' and sign in as " + str(cfg.get("email_id")) + ". Then re-run this script.")


def upsert_records(ctx, cfg):
    frappe = ctx.frappe
    dt, key = cfg["doctype"], cfg["match_field"]
    meta = require_doctype(frappe, dt)
    for rec in cfg["records"]:
        rec = clean(rec)
        check_fields(meta, rec.keys(), dt)
        existing = frappe.db.get_value(dt, {key: rec[key]}, "name")
        if existing:
            log(f"  = {dt} '{rec[key]}' exists ({existing}), skipped")
            continue
        doc = frappe.get_doc({"doctype": dt, **rec})
        doc.insert(ignore_permissions=True)
        log(f"  + {dt} '{rec[key]}' created")


# ---------- modes ----------
def run_inspect(frappe, extra):
    names = DEFAULT_INSPECT + extra
    for pattern in ("%Attach%", "%S3%"):
        names += frappe.get_all("DocType", filters={"name": ["like", pattern], "istable": 0}, pluck="name")
    seen = set()
    for dt in names:
        if dt in seen:
            continue
        seen.add(dt)
        if not frappe.db.exists("DocType", dt):
            log(f"\n## {dt}: NOT FOUND")
            continue
        meta = frappe.get_meta(dt)
        log(f"\n## {dt}   (single={meta.issingle}, module={meta.module})")
        for df in meta.fields:
            if df.fieldtype in LAYOUT_TYPES:
                continue
            opts = ""
            if df.options and df.fieldtype in ("Link", "Table", "Table MultiSelect", "Select", "Dynamic Link"):
                opts = "  -> " + str(df.options).replace("\n", " | ")
            req = " *required" if df.reqd else ""
            log(f"  {df.fieldname:38} {df.fieldtype:18} {df.label or ''}{req}{opts}")


def run_apply(frappe, cfg, dry_run, new_secret=False):
    ctx = Ctx(frappe, cfg, dry_run, new_secret)
    steps = [
        ("1.1 Website Settings", "website_settings", lambda c: set_single(ctx, "Website Settings", c)),
        ("1.2 System Settings", "system_settings", lambda c: set_single(ctx, "System Settings", c)),
        ("1.3 Navbar Settings", "navbar_settings", lambda c: set_single(ctx, "Navbar Settings", c)),
        ("1.3 Navbar help menu", "navbar_help", lambda c: navbar_help(ctx, c)),
        ("1.3 Core Apps", "core_apps", lambda c: core_apps(ctx, c)),
        ("2   Attachment List", "attachment_list", lambda c: upsert_records(ctx, c)),
        ("2.2 S3 File Attachment", "s3_settings", lambda c: set_single(ctx, c["doctype"], c["values"])),
        ("3   Connected App (Microsoft 365)", "connected_app", lambda c: connected_app(ctx, c)),
        ("3   Email Account (ICEGATE mailbox)", "email_account", lambda c: email_account(ctx, c)),
    ]
    for title, key, fn in steps:
        section = cfg.get(key)
        log(f"\n[{title}]")
        if not section:
            log("  (not in config, skipped)")
        elif section.get("enabled") is False:
            log("  (enabled: false, skipped)")
        else:
            fn(section)


def main():
    p = argparse.ArgumentParser(description="Canofy site setup automation")
    p.add_argument("mode", choices=["inspect", "apply"])
    p.add_argument("--site", required=True)
    p.add_argument("--config", help="customer JSON config (apply mode)")
    p.add_argument("--dry-run", action="store_true", help="check and print, then roll back")
    p.add_argument("--doctype", action="append", default=[], help="extra doctype to inspect (repeatable)")
    p.add_argument("--new-secret", action="store_true", help="prompt for a new Connected App client secret")
    a = p.parse_args()

    if not os.path.isfile(os.path.join(os.getcwd(), a.site, "site_config.json")):
        sys.exit("ERROR: run this from ~/frappe-bench/sites  (site folder not found in the current directory)")
    if a.mode == "apply" and not a.config:
        sys.exit("ERROR: apply needs --config <file.json>")

    import frappe

    frappe.init(site=a.site, sites_path=".")
    frappe.connect()
    frappe.set_user("Administrator")
    try:
        if a.mode == "inspect":
            run_inspect(frappe, a.doctype)
        else:
            with open(os.path.expanduser(a.config)) as f:
                cfg = json.load(f)
            log(f"Site: {a.site}   Customer: {cfg.get('customer_code', '?')}   {'DRY RUN' if a.dry_run else 'APPLY'}")
            run_apply(frappe, cfg, a.dry_run, a.new_secret)
            if a.dry_run:
                frappe.db.rollback()
                log("\nDRY RUN finished - nothing was saved.")
            else:
                frappe.db.commit()
                frappe.clear_cache()
                log("\nAPPLIED and committed. Hard-refresh the browser (Ctrl+Shift+R).")
    except SystemExit:
        frappe.db.rollback()
        raise
    except Exception:
        frappe.db.rollback()
        log("\nFAILED - all changes rolled back.")
        raise
    finally:
        frappe.destroy()


if __name__ == "__main__":
    main()

B. bel_setup.json

Customer config for Belimo. Copy it per customer. The email sections are disabled by default.

{
  "_about": "Canofy site setup config for BEL (Belimo Automation India). Keys starting with _ are comments.",
  "customer_code": "BEL",
  "branding_dir": "~/canofy_setup/branding",
  "website_settings": {
    "_sheet": "1.1 Website Settings",
    "app_name": "Canofy",
    "app_logo": "@Logo 150x150.png",
    "banner_image": "@Canofy logo 150px.png",
    "splash_image": "@Canofy logo 150px.png",
    "favicon": "@Canofy Favicon1.png"
  },
  "system_settings": {
    "_sheet": "1.2 System Settings (2FA is enabled later, after email works)",
    "enable_onboarding": 1,
    "login_with_email_link": 0,
    "session_expiry": "00:30",
    "deny_multiple_sessions": 1,
    "allow_login_using_user_name": 1,
    "allow_consecutive_login_attempts": 3,
    "allow_login_after_fail": 120
  },
  "navbar_settings": {
    "_sheet": "1.3 Navbar Settings",
    "app_logo": "@Logo 150x150.png"
  },
  "navbar_help": {
    "_sheet": "1.3 Navbar help dropdown",
    "hide": [
      "About",
      "Frappe Support"
    ],
    "add": [
      {
        "item_label": "About Canofy",
        "item_type": "Action",
        "action": "frappe.ui.toolbar.show_custom_about()"
      }
    ]
  },
  "core_apps": {
    "_sheet": "1.3 Core Apps. Title, version and branch are read from the bench automatically.",
    "doctype": "Core Apps",
    "table_field": "apps",
    "apps": [
      "canofy_tcs",
      "s3_file_storage",
      "custom_belimo"
    ]
  },
  "attachment_list": {
    "_sheet": "2 Attachment List (doctype 'Attachments', child 'Attachments Required For')",
    "doctype": "Attachments",
    "match_field": "attachment_name",
    "records": [
      {
        "attachment_name": "AWB_HBL",
        "attachments_required_for": [
          {
            "doctype_name": "Bill of Entry"
          },
          {
            "doctype_name": "FTWZ BOE Import"
          },
          {
            "doctype_name": "FTWZ DTA Sale"
          }
        ]
      },
      {
        "attachment_name": "Invoice",
        "attachments_required_for": [
          {
            "doctype_name": "Bill of Entry"
          },
          {
            "doctype_name": "FTWZ BOE Import"
          },
          {
            "doctype_name": "FTWZ DTA Sale"
          }
        ]
      },
      {
        "attachment_name": "Bill of Entry",
        "attachments_required_for": [
          {
            "doctype_name": "Bill of Entry"
          },
          {
            "doctype_name": "FTWZ BOE Import"
          },
          {
            "doctype_name": "FTWZ DTA Sale"
          }
        ]
      },
      {
        "attachment_name": "Packing List",
        "attachments_required_for": [
          {
            "doctype_name": "Bill of Entry"
          },
          {
            "doctype_name": "FTWZ BOE Import"
          },
          {
            "doctype_name": "FTWZ DTA Sale"
          }
        ]
      },
      {
        "attachment_name": "Attachment Type",
        "attachments_required_for": [
          {
            "doctype_name": "Bill of Entry"
          }
        ]
      }
    ]
  },
  "s3_settings": {
    "_sheet": "2.2 S3 File Settings. No AWS keys: the instance role provides access.",
    "doctype": "S3 File Settings",
    "values": {
      "bucket_name": "bel.canofy.tech",
      "region_name": "ap-south-1",
      "folder_name": "BEL",
      "signed_url_expiry_time": 300,
      "delete_file_from_cloud": 0,
      "download_file_without_signed_url": 0,
      "ignore_mapped_doctypes": 1
    }
  },
  "connected_app": {
    "enabled": false,
    "_note": "Set to true only if email is configured by script instead of the manual SOP path (phase 24).",
    "_sheet": "3 Email linking - Connected App. Fixed name = matches the Entra redirect URI. Secret is asked at a hidden prompt.",
    "name": "canofy-bel-m365",
    "provider_name": "Microsoft 365",
    "openid_configuration": "https://login.microsoftonline.com/<TENANT_ID>/v2.0/.well-known/openid-configuration",
    "client_id": "<CLIENT_ID of Canofy-CODE app>",
    "scopes": [
      "https://outlook.office.com/IMAP.AccessAsUser.All",
      "https://outlook.office.com/SMTP.Send",
      "offline_access"
    ]
  },
  "email_account": {
    "enabled": false,
    "_note": "Set to true only if email is configured by script instead of the manual SOP path (phase 24).",
    "_sheet": "3 Email Account - mirrors MIL 'No Reply Mil'. INBOX -> Icegate Intimation is the ICEGATE filtering.",
    "email_account_name": "No Reply Bel",
    "email_id": "no-reply-bel@canofy.tech",
    "auth_method": "OAuth",
    "connected_app": "canofy-bel-m365",
    "connected_user": "Administrator",
    "use_imap": 1,
    "use_ssl": 1,
    "use_starttls": 0,
    "email_server": "outlook.office365.com",
    "incoming_port": "993",
    "attachment_limit": 50,
    "email_sync_option": "UNSEEN",
    "initial_sync_count": "250",
    "append_emails_to_sent_folder": 0,
    "create_contact": 1,
    "enable_automatic_linking": 0,
    "use_tls": 1,
    "use_ssl_for_outgoing": 0,
    "smtp_server": "smtp.office365.com",
    "smtp_port": "587",
    "always_use_account_email_id_as_sender": 1,
    "always_use_account_name_as_sender_name": 1,
    "send_unsubscribe_message": 1,
    "track_email_status": 1,
    "no_smtp_authentication": 0,
    "imap_folder": [
      {
        "folder_name": "INBOX",
        "append_to": "Icegate Intimation"
      }
    ],
    "enable_after_authorize": {
      "enable_incoming": 1,
      "default_incoming": 1,
      "enable_outgoing": 1,
      "default_outgoing": 1
    }
  }
}

C. canofy_entra_app.sh

Optional scripted alternative to phase 24.2, for an operator with Azure CLI access to the Canofy tenant: bash canofy_entra_app.sh BEL bel.canofy.tech.

#!/usr/bin/env bash
# ---------------------------------------------------------------------------
# Canofy - Microsoft Entra app registration for a customer's mail integration
# Matches the existing pattern (Canofy-COS/BOS/EAP/LAX/SLC):
#   single tenant, Microsoft Graph delegated: IMAP.AccessAsUser.All,
#   offline_access, SMTP.Send, User.Read; Web redirect = Frappe Connected App
#   callback; client secret "Canofy-<CODE>-secret".
#
# Run in Azure Cloud Shell (Bash) or any machine with Azure CLI, signed in as
# an admin of the Canofy tenant:
#   az login --allow-no-subscriptions      # not needed in Azure Cloud Shell
#   bash canofy_entra_app.sh BEL bel.canofy.tech
#   bash canofy_entra_app.sh BEL bel.canofy.tech --admin-consent   # optional
#
# Safe to re-run: reuses the app if it exists, only adds what is missing.
# A NEW secret is created only with --new-secret (or on first creation).
# ---------------------------------------------------------------------------
set -euo pipefail

CODE="${1:?Usage: $0 <CODE> <site-domain> [--admin-consent] [--new-secret]}"
SITE="${2:?Usage: $0 <CODE> <site-domain> [--admin-consent] [--new-secret]}"
shift 2
ADMIN_CONSENT=0; NEW_SECRET=0
for a in "$@"; do
  case "$a" in
    --admin-consent) ADMIN_CONSENT=1 ;;
    --new-secret)    NEW_SECRET=1 ;;
    *) echo "Unknown option: $a"; exit 1 ;;
  esac
done

CODE_UP=$(echo "$CODE" | tr '[:lower:]' '[:upper:]')
CODE_LO=$(echo "$CODE" | tr '[:upper:]' '[:lower:]')
APP_NAME="Canofy-${CODE_UP}"
SECRET_NAME="Canofy-${CODE_UP}-secret"
CONNECTED_APP="canofy-${CODE_LO}-m365"          # fixed Connected App name in Frappe
REDIRECT_URI="https://${SITE}/api/method/frappe.integrations.doctype.connected_app.connected_app.callback/${CONNECTED_APP}"
SECRET_DAYS="${SECRET_DAYS:-180}"               # override: SECRET_DAYS=365 bash ...
GRAPH="00000003-0000-0000-c000-000000000000"    # Microsoft Graph appId (fixed)
SCOPES=("IMAP.AccessAsUser.All" "offline_access" "SMTP.Send" "User.Read")

TENANT_ID=$(az account show --query tenantId -o tsv)
echo "Tenant      : $(az account show --query tenantDisplayName -o tsv 2>/dev/null || true) ($TENANT_ID)"
echo "App         : $APP_NAME"
echo "Redirect URI: $REDIRECT_URI"
echo

# 1) App registration (reuse if it exists)
APP_ID=$(az ad app list --display-name "$APP_NAME" --query "[0].appId" -o tsv)
CREATED=0
if [ -z "$APP_ID" ]; then
  APP_ID=$(az ad app create --display-name "$APP_NAME" --sign-in-audience AzureADMyOrg \
            --web-redirect-uris "$REDIRECT_URI" --query appId -o tsv)
  CREATED=1
  echo "+ created app registration $APP_NAME"
else
  echo "= app registration exists"
  # make sure the redirect URI is present (keep any existing ones)
  EXISTING=$(az ad app show --id "$APP_ID" --query "web.redirectUris" -o tsv | tr '\t' '\n')
  if ! echo "$EXISTING" | grep -qxF "$REDIRECT_URI"; then
    # shellcheck disable=SC2086
    az ad app update --id "$APP_ID" --web-redirect-uris $EXISTING "$REDIRECT_URI"
    echo "+ added redirect URI"
  else
    echo "= redirect URI already set"
  fi
fi

# 2) Delegated Microsoft Graph permissions (IDs looked up, not hard-coded)
CURRENT=$(az ad app show --id "$APP_ID" \
  --query "requiredResourceAccess[?resourceAppId=='$GRAPH'].resourceAccess[].id" -o tsv | tr '\t' '\n')
for S in "${SCOPES[@]}"; do
  SID=$(az ad sp show --id "$GRAPH" --query "oauth2PermissionScopes[?value=='$S'].id | [0]" -o tsv)
  if [ -z "$SID" ]; then echo "ERROR: Graph scope $S not found"; exit 1; fi
  if echo "$CURRENT" | grep -qxF "$SID"; then
    echo "= permission $S already present"
  else
    az ad app permission add --id "$APP_ID" --api "$GRAPH" --api-permissions "${SID}=Scope" >/dev/null 2>&1
    echo "+ added permission $S (delegated)"
  fi
done

# 3) Service principal (the "Enterprise application" entry used for consent)
if [ -z "$(az ad sp list --filter "appId eq '$APP_ID'" --query "[0].id" -o tsv)" ]; then
  az ad sp create --id "$APP_ID" >/dev/null
  echo "+ created service principal"
fi

# 4) Optional tenant-wide admin consent (only if user consent is blocked)
if [ "$ADMIN_CONSENT" = "1" ]; then
  sleep 10   # give Entra a moment to see new permissions
  az ad app permission admin-consent --id "$APP_ID"
  echo "+ admin consent granted"
fi

# 5) Client secret (only on first creation or with --new-secret)
SECRET=""
if [ "$CREATED" = "1" ] || [ "$NEW_SECRET" = "1" ]; then
  END_DATE=$(date -u -d "+${SECRET_DAYS} days" +%Y-%m-%dT%H:%M:%SZ)
  SECRET=$(az ad app credential reset --id "$APP_ID" --append --display-name "$SECRET_NAME" \
            --end-date "$END_DATE" --query password -o tsv 2>/dev/null)
  echo "+ created client secret '$SECRET_NAME' (expires $END_DATE)"
else
  echo "= secret not changed (use --new-secret to create one)"
fi

cat <<EOF

================ VALUES FOR THE FRAPPE CONNECTED APP ================
Connected App name : $CONNECTED_APP
Client ID          : $APP_ID
OpenID config URL  : https://login.microsoftonline.com/$TENANT_ID/v2.0/.well-known/openid-configuration
Redirect URI       : $REDIRECT_URI
EOF
if [ -n "$SECRET" ]; then
cat <<EOF
Client SECRET      : $SECRET
   ^ shown ONCE. Save it in the password manager now, with its expiry date.
     Do not paste it into chat, tickets or files.
EOF
fi
echo "====================================================================="

D. ocr_add_customer.py

Optional scripted alternative to phase 25.3. Run on the OCR server as ubuntu: /usr/bin/python3 ocr_add_customer.py --app-dir /home/ubuntu/canofy_OCR --code BEL --ip 172.31.10.27 --bucket bel.canofy.tech --site bel.canofy.tech --ocr-url http://172.31.35.206:5000/generate_ocr --dry-run.

#!/usr/bin/env python3
"""
Canofy OCR - add a customer (ERP server) to property.json

Run ON THE OCR SERVER, with the same Python the OCR service uses:

  /usr/bin/python3 ocr_add_customer.py --code BEL --ip 172.31.10.27 --bucket bel.canofy.tech --site bel.canofy.tech --dry-run
  /usr/bin/python3 ocr_add_customer.py --code BEL --ip 172.31.10.27 --bucket bel.canofy.tech --site bel.canofy.tech

What it does
  * reads ENCRYPTION_KEY from ~/canofy_OCR/.env (never typed, never printed)
  * generates a random OCR username + password for this customer
  * Fernet-encrypts them exactly like encrypt.py, and checks they decrypt
  * refuses to overwrite an existing IP entry (use --replace to rotate)
  * backs up property.json, writes the new entry, validates the JSON
  * prints the 3 'bench set-config' commands to run on the ERP server
    (ocr_username / ocr_password base64-encoded, ocr_url)
"""
import argparse
import base64
import json
import os
import secrets
import shutil
import socket
import sys
from datetime import datetime

from cryptography.fernet import Fernet


def read_env_key(env_path):
    with open(env_path) as f:
        for line in f:
            line = line.strip()
            if line.startswith("ENCRYPTION_KEY="):
                return line.split("=", 1)[1].strip().strip('"').strip("'")
    sys.exit(f"ERROR: ENCRYPTION_KEY not found in {env_path}")


def own_private_ip():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        s.connect(("10.255.255.255", 1))
        return s.getsockname()[0]
    finally:
        s.close()


def main():
    home = os.path.expanduser("~")
    p = argparse.ArgumentParser()
    p.add_argument("--code", required=True, help="customer code, e.g. BEL")
    p.add_argument("--ip", required=True, help="ERP server PRIVATE IP, e.g. 172.31.10.27")
    p.add_argument("--bucket", required=True, help="customer S3 bucket, e.g. bel.canofy.tech")
    p.add_argument("--site", required=True, help="Frappe site name, e.g. bel.canofy.tech")
    p.add_argument("--app-dir", default=os.path.join(home, "canofy_OCR"))
    p.add_argument("--ocr-url", help="default: http://<this server private IP>:5000/generate_ocr")
    p.add_argument("--replace", action="store_true", help="rotate credentials of an existing IP entry")
    p.add_argument("--dry-run", action="store_true")
    a = p.parse_args()

    prop_path = os.path.join(a.app_dir, "property.json")
    env_path = os.path.join(a.app_dir, ".env")
    for path in (prop_path, env_path):
        if not os.path.isfile(path):
            sys.exit(f"ERROR: not found: {path}  (use --app-dir if the OCR app lives elsewhere)")

    fernet = Fernet(read_env_key(env_path).encode())
    with open(prop_path) as f:
        props = json.load(f)
    if not isinstance(props, dict):
        sys.exit("ERROR: property.json is not a JSON object")

    # Report existing entries (IP -> bucket) and sanity-check their format
    print(f"property.json has {len(props)} entries:")
    for ip, entry in props.items():
        print(f"  {ip:16} -> {entry.get('BUCKET NAME')}")
        u = str(entry.get("username", ""))
        if u.startswith("b'") or u.startswith('b"'):
            print("    NOTE: this entry stores values as b'...' text; new entry uses plain token text")
        try:
            fernet.decrypt(u.strip("b'\"").encode())
        except Exception:
            print("    WARNING: this entry does not decrypt with the current key")

    if a.ip in props and not a.replace:
        sys.exit(f"ERROR: {a.ip} already exists (bucket {props[a.ip].get('BUCKET NAME')}). Use --replace to rotate.")
    for ip, entry in props.items():
        if entry.get("BUCKET NAME") == a.bucket and ip != a.ip:
            print(f"WARNING: bucket {a.bucket} is already used by {ip}")

    username = f"{a.code.lower()}_ocr_{secrets.token_hex(3)}"
    password = secrets.token_urlsafe(24)
    enc_user = fernet.encrypt(username.encode()).decode()
    enc_pass = fernet.encrypt(password.encode()).decode()
    assert fernet.decrypt(enc_user.encode()).decode() == username
    assert fernet.decrypt(enc_pass.encode()).decode() == password

    new_entry = {"BUCKET NAME": a.bucket, "username": enc_user, "password": enc_pass}
    ocr_url = a.ocr_url or f"http://{own_private_ip()}:5000/generate_ocr"

    print(f"\nNew entry for {a.ip}: BUCKET NAME={a.bucket}, username/password = Fernet tokens (verified)")
    if a.dry_run:
        print("\nDRY RUN - property.json not changed.")
        return

    backup = f"{prop_path}.bak.{datetime.now():%Y%m%d-%H%M%S}"
    shutil.copy2(prop_path, backup)
    props[a.ip] = new_entry
    tmp = prop_path + ".tmp"
    with open(tmp, "w") as f:
        json.dump(props, f, indent=4)
    with open(tmp) as f:
        json.load(f)  # validate before replacing
    os.replace(tmp, prop_path)
    print(f"Backup : {backup}")
    print(f"Written: {prop_path}  ({len(props)} entries)")

    b64 = lambda s: base64.b64encode(s.encode()).decode()
    print("\n=========== RUN THESE ON THE ERP SERVER (as frappe, in ~/frappe-bench) ===========")
    print(f"bench --site {a.site} set-config ocr_username {b64(username)}")
    print(f"bench --site {a.site} set-config ocr_password {b64(password)}")
    print(f"bench --site {a.site} set-config ocr_url {ocr_url}")
    print("===================================================================================")
    print("These are credentials: copy them straight into the ERP session. Do not paste them")
    print("into chat, tickets or email. Then restart the OCR service:")
    print("  sudo systemctl restart ocr_app.service && sudo systemctl status ocr_app.service --no-pager")


if __name__ == "__main__":
    main()

E. Screenshot register

Figures embedded from the Belimo onboarding, with sensitive identifiers masked, and screenshots still to capture for the SOP.

FigureContentStatus
1SS-05a: the Marketplace "Subscribe" page for Ubuntu 22.04. Close it and use the Canonical Community AMI instead.embedded
2SS-06: instance summary after launch: Running, t3a.medium, role EC2-Canofy-BEL-SSM-Prod, auto-assigned public IP.embedded
3SS-07: volume vol-0cc1d95a7394d274a: 20 GiB gp3, 3000 IOPS, Encrypted with aws/ebs. Note the Customer tag here still holds a wrong value, which was corrected afterwards.embedded
4SS-08: the Elastic IP 3.6.128.118 now replaces the auto-assigned address.embedded
5SS-10: first SSM session checks: ssm-user, Ubuntu 22.04.5, agent active, outbound IP 3.6.128.118.embedded
6SS-11: bucket properties: versioning enabled, SSE-S3 with Bucket Key, SSE-C blocked. The tag key "customer" must be "Customer".embedded
7SS-17: role identity and S3 upload, list and delete, with no access keys on the server.embedded
8Before: the saved policy has no bel.canofy.tech lines, so OCR is blocked by the Deny statement.embedded
9SS-13: after: BEL lines 20–21 (Allow) and 40–41 (Deny/NotResource), "updated" banner, Errors 0.embedded
10The suggestion markers on lines 22–23 come from the MIL trailing-slash typo, not from the new entries.embedded
11Existing policies: an enabled default 14-day policy for all volumes, and the enabled 60_day_production_instances policy.embedded
12SS-15: target tags of the 60-day policy. Every customer uses key "Name" with the volume name. The "BEL" key entry is the mistake to avoid.embedded
13SS-11a: hostname bel-prod, timezone Asia/Kolkata (IST).embedded
14SS-11b: 4.0 GB swap active on /swapfile after the reboot.embedded
15SS-SW-01: frappe in the sudo group, Python 3.11.16, Node v20.20.2, yarn 1.22.22, MariaDB 10.6.23, Redis 6.0.16, nginx 1.18.0, wkhtmltopdf 0.12.6.1 with patched Qt.embedded
16SS-DB-01: the config written with printf (one setting per line) and MariaDB active (running). The first lines show the earlier merged "256M[mysql]" that broke the service.embedded
17SS-DB-02: secure installation done; utf8mb4, 127.0.0.1, 1 GB buffer pool; root password login works; system Redis inactive.embedded
18SS-19: bench initialised, frappe 15.72.5, Python 3.11.16.embedded
19Reference versions on MIL: frappe 15.72.5, canofy_tcs 2.2.1, canofy_ddit 1.0.2, s3_file_storage 0.0.1, custom_mil_power (MIL only), Python 3.10.12, Node 18.20.8.embedded
20App remotes on MIL: canofy_TCS on development (e5c1cf96), canofy_ddit main (v1.0.1 tag, 782a128), priv-frappe-s3-file-storage main (163256d).embedded
21Without credentials, git reports "could not read Username", which means the repo is private.embedded
22SS-21: versions and commits after download. canofy_tcs on development came in at 2.3.0, newer than MIL's 2.2.1, which is expected when following a branch.embedded
23Settings → Secrets and variables → Actions is NOT where deploy keys go. Deploy keys need admin rights on the repo.embedded
24canofy_tcs installed: DocTypes 100%, Address customisations, Dashboard.embedded
25SS-22: apps on the BEL site: frappe 15.72.5 (pinned tag), canofy_tcs 2.3.0 development, s3_file_storage 0.0.1, custom_belimo 1.2.3. canofy_ddit stays in the bench only.embedded
26SS-23: sudoers set up, setup production complete, all 7 processes RUNNING, HTTP 200.embedded
27SS-25: log format file, nginx test OK, HTTPS 200, HTTP 301, renewal dry-run succeeded.embedded
28SS-24: login page served at https://bel.canofy.tech, before branding.embedded
29SS-26: scheduler enabled; bench's 6-hourly local backup plus the 01:30 S3 job; test backup (330.2 KiB) in _backups/.embedded
30SS-AUTO-01: Belimo apply output: logos placed as local public files, settings changed, help menu, Core Apps, 5 Attachments, S3 File Settings, "APPLIED and committed".embedded
31Signed in as no-reply-bel: Canofy Technologies tenant, display name "Canofy Support [Belimo]", no roles needed.embedded
32SS-34: Canofy-BEL overview (IDs masked): single tenant, 1 web redirect URI, Activated.embedded
33SS-35: Microsoft Graph (4), all delegated, admin consent not required.embedded
34SS-36: Canofy-BEL-secret, expires 25 Sep 2027 (value and secret ID masked).embedded
35The service runs gunicorn from /home/ubuntu/canofy_OCR, so the live file is /home/ubuntu/canofy_OCR/property.json.embedded
36Several copies exist (backups and a stray /home/property.json). Edit only the copy in the service's working directory.embedded
37ssm-user can't list /home/ubuntu; switch with sudo -iu ubuntu first.embedded
38MariaDB failed: heredoc lines merged by the SSM terminal.embedded
39bench init stuck at a sudo password prompt for frappe.embedded
40bench init traceback at sudo supervisorctl status, with the rollback prompt.embedded
41The bench folder wasn't found on another server: always locate it first with find … -name apps.txt.embedded
42Loop variables are lost inside sudo -iu … bash -c '…': switch user first.embedded
43bench get-app failed authentication: a password or failed paste instead of a token.embedded
44Expected "no such group" warning after get-app, before production mode.embedded
45new-site failed: No module named pkg_resources (Dropbox Settings import).embedded
46install-app failed: Row size too large on tabBOE Items.embedded
47install-app custom_belimo: InvalidRemoteException.embedded
48Traceback showing the required_apps lookup on GitHub (org erpnext → 404).embedded
49bench doctor: Redis 11000 refused before production mode.embedded
50The same Redis error after a setup production that didn't complete.embedded
51502 Bad Gateway: nginx linked, but /etc/supervisor/conf.d is empty.embedded
52supervisorctl status shows no processes.embedded
53After the manual supervisor link and chown: 7 processes RUNNING, HTTP 200.embedded
54setup sudoers failed: certbot not installed.embedded
55Certificate issued, but nginx -t fails with unknown log format "main".embedded
56Only the logos reached S3, in the wrong folder, so the script was missing.embedded
57Dry run: System Settings, Navbar and Core Apps OK; Attachments failed on FTWZ names (rolled back).embedded
58Apply failed in the s3_file_storage hook before the S3 settings existed (rolled back).embedded
59grep -c placed = 0: an old script was still in S3.embedded
60The OCR policy editor with BEL lines added. Without Next → Save changes, this isn't saved.embedded
61set-config -p server_script_enabled true raises a malformed node error; use 1.embedded
—SS-04: Security group inbound rules showing only 80 and 443to capture
—SS-12: Bucket Permissions tab with Block Public Access on and the policy savedto capture
—SS-14: Both lifecycle rules listed on the Management tabto capture
—SS-16: GoDaddy DNS record list showing the bel A recordto capture
—SS-27: Login page and desk with Canofy brandingto capture
—SS-37: Saved Connected App, and the Entra Authentication page with the matching redirect URIto capture
—SS-38: Icegate Intimation record created from the test emailto capture